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Abstract — Robotic grasping is traditionally approached as a 
pure planning problem that assumes a priori knowledge of the 
target object's geometry. However, we would like our robot 
to be able to robustly grasp objects with which it has no 
prior experience. Our approach is to use haptic information to 
drive a search process for appropriate finger contact locations. 
Given a cost function that is based on the total force and 
moment applied to the object by the set of contacts, and 
simple assumptions about the local surface geometry, this search 
process can be formulated as one of gradient descent on the 
cost function. Prior work in this area has assumed that the 
surface of the object local to the contact is either flat or convex. 
However, when the surface is concave, the search process, in 
fact, ascends the cost function. Here, we propose a switching 
controller approach that estimates the local curvature of the 
object over multiple contacts. This information is then used to 
switch between one of two methods of estimating the gradient of 
the cost function. While this new approach shows comparable 
performance to the original when faced with objects containing 
only flat or convex surfaces, the new algorithm performs 
substantially better when objects contain concave surfaces. 



I. INTRODUCTION 
Robotic grasping approaches have traditionally relied on a 
priori knowledge of the geometry of the object being grasped 
or on estimates of the geometry based on visual or range 
inputs (e.g., Bekey et a/.[l], and Borst et flZ.[3], Miller et 
a/. [12]). In either case, the modeled geometry is used to 
assess the quality of many potential grasps before one is 
selected for execution. In an alternative approach, one could 
make minimal a priori assumptions about the geometry of 
the object being grasped, and instead rely on haptic feedback 
to direct the tactile exploration of an object until a suitable 
grasp is found. Teichmann and Mishra [20], and subsequently 
Coelho and Grupen [4], introduced methods in which the 
local surface normal for each of several contacts was first 
estimated. Based on this information, contact displai 
were computed that followed the negative gradient c 
function. The cost functions were such that their 
corresponded to a quality grasp of the object. In the former 
case, this cost function was based on the area of the triangle 
formed by three contact points. In the latter case, two cost 
functions were used: one that described the net force apphed 
to the object by the set of contacts, and another that described 
t applied by the s£ 
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In the case of Coelho and Grupen [4], and in subsequent 
work by Piatt et al. [15], [14], the gradients of the cost func- 
tions were estimated by making simple assumptions about 
the object's surface properties, including local geometry. 
First of all, the contacts were assumed to be frictionless. 
In the case of the force cost function, the surface was also 
assumed to be locally convex, and in particular, that it was 
a unit sphere. In the case of the moment cost function, the 
surface was assumed to be planar. The force controller and 
moment controller were defined such that each contact was 
displaced so that it followed the negative gradient of the 
corresponding cost function. Piatt et al. combined the actions 
of the two controllers through a nuUspace operation that 
favored the actions of the force controller over those of the 
moment controller. 

This work has showed promise in enabling a grasping sys- 
tem to interact with objects of unmodeled geometries [14], 
[17]. However, the current approach suffers when the ob- 
ject's surface differs substantially from assumption of local 
convexity. Specifically, when the surface is concave, the force 
controller will drive the contact in a direction that serves to 
increase the net force rather than decrease it. This behavior 
substantially limits the class of objects that can be addressed 
by this grasp search approach. 

The challenge, therefore, is how to appropriately address 
objects that contain concavities. Park and Starr [13] describe 
a grasp synthesis method for polygons of known shape 
that considers as potential contact locations convex and 
concave vertices. Funahashi et al. [6] analyzed the stability 
of grasps involving fingers with controllable stiffness and 
either concave object surfaces or fingers. However, they do 
not address the grasp synthesis step. In this paper, we address 
the concavity issue by introducing a second force controller 
that makes the assumption of local concavity. A meta-level 
controller then switches, on a per-contact basis, between 
the convex and concave control actions as a function of 
the estimated curvature of the object. A set of simulation 
experiments are used to demonstrate the utility of the hybrid 
convex/concave controller over the convex-only controller. 

II. METHODS 

The goal is to find a set of contact locations on an object 

such that a set of non-zero normal forces applied at the 

contacts result in a net force and moment applied to the 

object of zero. In the presence of soft contacts and friction. 



a set of contacts that satisfy these criteria ensures wrench 
closure, allowing the object to be squeezed at the contacts 
without accelerating it [18], [16]. 

Our approach to finding this set of contact locations 
involves a gradient descent search that requires the fingers 
to haptically explore the object. At each step of the search, 
contact is made with the object by each of the finger tips. 
Given an estimate of each of the contact locations and surface 
normals, the net force and moment applied to the object 
is estimated (assuming unit force applied at each contact). 
Given assumptions about the local geometry of each surface, 
contact displacements are estimated that are intended to 
reduce the net force and moment applied to the object. In the 
subsequent sections, we describe the process of computing 
these contact displacements. 

A. Force Controller 

The task of the /orce controller is to reduce the total force 
applied to the object by the set of contacts. Following Coelho 
and Grupen [4], the error function for this controller is: 




Fig. 1 . Force control for convex object, fi is tlie current force exerted by 
tlie finger and f2 is tlie future force exerted by tlie finger, fe is tlie external 
force on the object. 



where /? > is a stiffness coefficient with units of N/m. The 
second partial derivative of equation 3 can be approximated 
as follows: 



where f„ is a column vector describing the net force applied 
by the set of contacts. This is computed as follows: 



fn = f + fe, 



where f is the unit force applied by the contact of interest, 
and is normal to the object surface at the contact point, and 
fe is the sum of the remaining forces, including the other 
contacts and any modeled (but uncontrolled) external forces. 
Note that one can assume a force vector of arbitrary length 
to express the differential roles played by various contacts in 
a grasp. However, we do not exploit this flexibility for the 
purposes of this work. 

Let X denote the Cartesian location of the contact in R^ . 
Following each probe of the object, the contact is displaced 
along a direction that reduces ej. This direction is computed 
by first estimating the gradient of the error function: 
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where I is a 3 by 3 identity n 
4, and 6, we have: 
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L. Combining equations 3, 
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Finger displacement will follow the negative gradient of 
the error function. Therefore, the displacement will be in the 
same direction as f„, subject to maintaining contact with the 
object's surface. 

The alternative to assuming a locally convex surface is 
to assume a concave one. For the concave controller, we as- 
sume that the contact is located on the inside of a unit sphere. 
This scenario is illustrated in Fig. 2. As with the previous 
case, we assume a small displacement in the direction of 
Ax (the surface tangent), and then a small translation back 
to the surface, toward the origin of the sphere. In this case, 
the following holds: 



(3) 
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= f„ 



(4) 



I respect to contact loca- 
5 about the local surface 



dt 
The gradient of the force wi 
tion depends on our assumptio 
geometry. For the convex controller, we follow the spherical 
assumption of Coelho and Grupen [4]. This scenario is 
illustrated in Fig. 1. The current contact produces force fi. 
Given that the other contacts produce force fe, the total 
force appUed to the object is shown as f„. The finger tip 
is then displaced along Ax (the tangent to the sphere at the 
original contact), and then translated toward the origin of the 
sphere. For a small Ax, this latter translation is very small. 
Therefore: 



Combining equations 3, 4 and 9, we have: 
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The result is that the contact should be displaced along the 
direction opposite to f„. Note that this displacement should 
be done subject to maintaining some ideal force against 
the surface of the object (such that the object itself is not 
displaced). Note also that the direction of displacement given 
the concave assumption is in the opposite direction as the 
convex assumption. 





Fig. 2. Force control for concave object, fi is the current force exerted by 
the tinger and f2 is the future force exerted by the finger, fe is the external 
force on the object. 



Fig. 3. Moment control, f is the force exerted by the finger and fe is 
the external force on the object, "o" is the origin about which torque is 
measured, ri and ro denote the vector from current and future contact 
point to the origin, respectively. In current configuration, the net torque 
points inside the paper. 



) reduce the total 



B. Moment Controller 

The role of the moment controller 
moment applied to the object. The i 
function is defined as: 



where in„ is the net moment applied to the object. Here, 
net moment is composed of the moment due to the contact 
of interest (m), and the moment due to any other modeled 
forces external to 



The gradient of the error function with respect to contact 
displacement is as follows: 
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; the derivative of the contact-imposed 
moment with respect to contact displacement by following 
the planar surface assumption of Coelho and Grupen [4]. 
This situation is illustrated in Fig. 3. The current contact 
imposes force f at a position from which the moment 
reference point (o) is displaced by ri. Given a small contact 
displacement along the surface tangent of Ax: 
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= -fxm„. (16) 

The result is that the moment controller recommends 
contact displacements in the direction of f x in„. Note that 
in the general case, the choice of the moment reference 
point can affect the magnitude and sign of this recommended 
displacement. However, when the net force due to the set of 
contacts is zero, they form a couple. Under this condition, 
the net moment (and hence, the recommended displacement) 
becomes the same for any choice of reference point [11]. 

C. Concavity Detection 

Key to making use of the appropriate force control law (the 
convex or the concave form) is estimating the local surface 
curvature. One possible approach is to recover local curvature 
using a tactile or image array (e.g., Jiar et al. [9] and Lee 
& NichoUs [10]). In our case, we assume that a 6-axis load 
cell is embedded within the finger tip, from which one can 
infer the surface normal but not the local curvature [2]. 
Hence, it becomes necessary to integrate information over 
multiple probes of the object. At any given time, for the 
purposes of the force controller, a surface is assumed to be 
either concave or convex (with surfaces initially assumed to 



are 



be convex). Given two subsequent probes of an object, the 
assumed curvature is determined as follows: 



Af Ax < — «! assume convex 

Af^Ax > a2 assume concave 

otherwise no change 

where Af is the change the force exerted by the contact. Ax 
is the displacement of the contact, and ai > and q;2 > 
are switching parameters. These parameters are empirically 
selected so that the switching controller is sensitive enough 
to discover the concavities of the test objects, but is not dis- 
tracted substantially by uncertainties in the force estimation 
process (both are set to 0.5 Nm). 

D. Combining Force and Moment Control Actions 
Following Piatt et al, the displacement recommendations 

of both the force controller and the moment controller 
are combined into a single displacement [15], [14]. This 
combination gives the moment controller the opportunity 
to influence the motion of the contact as long as it does 
not interfere with the actions of the force controller [8], 
[7]. Specifically, the combined control action is computed 
as follows: 

(/> = (/>/+iV((/>/)0™, (17) 

where <j) is the composite displacement, 0/ and 
the recommended displacements by the force and 
controllers, respectively, and N{.) is a null space projection 

E. Switching Controller 

The sequential behavior required for the haptic search 
process is implemented using a finite state machine, as shown 
in Fig. 4. Each finger is controUed using one such FSM. The 
object is initiaUy assumed to be located such that flexing of 
the the fingers toward the palm will bring each finger into 
contact with the object. At each control step, the surface 
normal at the contact is used to estimate the local curvature of 
the surface and to recommend a displacement of the contact. 
The latter is computed using the combined force and moment 
controller recommendations (as described above), where the 
force controller is either of the type convex or concave. 
Displacement of the contact is implemented by drawing the 
finger away from the surface along the normal, translating 
the finger along the recommended displacement, and then 
placing the finger back onto the surface along the original 
surface normal. Under most conditions, the finger will again 
contact the surface, and the process will repeat from either 

The search process terminates under two conditions. First, 
if the net force and moment fall below a critical threshold, 
the search is considered to have terminated successfully (the 
Done state). Second, if the search is still ongoing after a 
specified period of time, the controller is considered to have 
terminated in an Error state (we say that the controller has 
"timed out"). One scenario in which this can happen is when 
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the controller has achieved an equilibrium state or is in a 
cycle in which the force or moment error does not satisfy 
the critical threshold. 

There are also a number of cases in which the finger 
does not successfully return to surface of the object after 
displacement is executed. This situation can happen if the 
local surface is not smooth with respect to the size of the 
contact displacement. For example, this can happen when 
is displaced off the corner of an object. This 
n also happen when there is an error in the 
estimation of the surface normal. Here, the recommended 
displacement can take the finger far away from the surface. 
In either case, the recover contact state is responsible for 
taking several heuristic approaches to bringing the finger 
back into contact with the object. Both problems are sensitive 
to the magnitude of the contact displacement for a single 
control step. Therefore, contact displacement magnitude is 
considered a parameter that must be selected appropriately 
(this is addressed in the Results section). 

F. Simulation Implementation 

All experiments were conducted within a custom simula- 
tion environment that is based on the Visualization Toolkit 
(VTK) package [19]. This toolkit aUows for the modeling 
of the surfaces of the robotic fingers and of the objects 
of interest. The surfaces are modeled as meshes; colUsion 
between two surfaces is detected as a collision between 
two mesh triangles. The force direction (and hence surface 
normal) is then determined by the orientation of the colliding 
triangles. Due to this approach, estimation of applied force 
exhibits a certain degree of variation, depending on small 
changes in orientation of the colUding triangles. 

The simulated robot used for the experiments described in 
this paper consists of two fingers. Each finger is four degrees 
of freedom: one adduction/abduction and three flexion. The 
fingers are mounted relative to one another such that they 
are able to oppose one-another. For these experiments, the 



hand is fixed in space. Hence, any objects are placed near the 
palm of the hand such that both fingers can initially touch 
the object through a simple flexion motion. 

III. EXPERIMENTAL RESULTS & EVALUATION 
The proposed controller is explicitly designed to properly 
explore objects with both convex and concave surfaces. 
Our experimental hypothesis is that when presented with an 
object containing a concave surface, the proposed controUer 
should perform better than one controller that only "expects" 
convex surfaces. However, when an object is composed of 
non-concave surfaces (convex or flat), the two controUers 
should perform identically. 

In addition to the choice of control algorithm, two other 
factors can significantly affect the performance of a con- 
troller: the contact displacement magnitude and orientation 
of the object relative to the fingers. The latter factor affects 
which of the object's surfaces are initially exposed to the 
fingers. We have therefore implemented (for most of the 
following experiments) a 3-factor analysis of algorithm, 
displacement magnitude, and object orientation. For each 
combination of factors, a sample size of iV = 20 is used. 
Although object orientation is a controUed factor, a smaU, 
random orientation is added to the object prior to initiation 
of the exploration process (uniform distribution: ±5 degrees). 
Two measures of performance were used: the total number of 
successful grasps (achieving the done state), and the running 
time of the controller (a controller "times out" when a probe 
occurs 400 seconds into the search). 

The implementation of the convex-only controller is the 
same as the switching controller (as described in Fig. 4). 
The only difference is that the concave controller state has 
been removed (and consequently, concavity detection is not 

A. Non-Concave Objects 

Two non-concave objects were used in this study: a sphere 
and a cube. Because the sphere is symmetric about aU 
rotations, we used a two-factor analysis (controllers (x2) 
and contact displacement magnitude (xl6)). Fig. 5 shows the 
percent of successful grasps for each algorithm and contact 
displacement magnitude ("step size"). Both controUers per- 
formed weU when step size was less than 10, but performance 
degraded with larger step sizes. This degradation is due to 
the fact that the larger step sizes tended to lead to instabilities 
around the equilibrium point. In addition, the larger step sizes 
could result in the finger "dropping off" of the object all 
together. 

Across all step sizes, the mean success for both controllers 
was 80%. Furthermore, a two-way ANOVA analysis on the 
total running time of the controller indicated a significant 
effect of step size (F = 40.56; p < 10^^), but no significant 
effect on choice of controller {F = .17; p > .68). There 
was, however, a significant interaction effect (F = 6.79; 
p < 10"^). This effect highlights the fact that the switching 
controller performed slightly worse for small step sizes. 
This effect is due to the controUer's inability to consistently 




estimate the curvature of the object when displacements 
are smaU. SpecificaUy, the "noise" in the contact normal 
estimation can sometimes dominate the curvature estimate 

The second non-concave object was a cube (Fig. 6). 
Across all step sizes and object orientations, the convex- 
only controller was successful 76.6% of the time, while 
the switching controUer was successful for only 73.6% of 
the trials. This difference was not significant according to 
a Fisher Exact Test (p > 0.09). A three-way ANOVA 
test on running time showed a significant effect on the 
average running time of step size (F = 29.4; p < 10^^) 
and on choice of controller (F = 6.78; p < .01). Mean 
running time for the convex-only controller was 204.75sec, 
while the switching controller required a mean of 219.32sec. 
The small additional cost in running time of the proposed 
controller was due to occasional misclassification of the 
surface curvature. This effect typically results in one or two 
control steps moving in the incorrect direction before the 
surface is correctly classified. 

B. Concave Objects 

Two objects containing concavities were used: a "concave 
cube," and a "capped cylinder" (see Fig. 6). The concave 
cube was constructed by subtracting hemispheres from each 
side of a cube. The result was a that a majority of its surface 
was concave, with only areas around the comers maintaining 
a flat surface. Most of the surface of the capped cylinder 
is convex, but one small region is concave. Depending on 
the orientation of the cylinder relative to the fingers, the 
initial contacts may "see" either both convex surfaces or one 
concave and one convex surface. 

Fig. 7 shows the performance of each controller given the 
step size and the object orientation. Orientations of 75 and 90 



degrees are such that both fingers are initially presented with 
a convex surface. For orientations of and 15 degrees, one 
finger falls well within the concave region. An orientation of 
45 degrees results in one finger landing on the edge between 
the concave and convex surface (with 30 and 60 degrees 
falling to either side of this sharp edge). 

Both controllers exhibited high levels of success for ori- 
), 75, and 90 degrees (Fig. 7a,b). However, 
s of the convex-only controller suffers dramatically 
when presented with the concave surface (orientations of 0, 
15, and 30 degrees). In contrast, the switching controller ( 7b) 
performed well for these orientations, most dramatically for 
and 15 degrees. However, this performance dropped off as 
the step size increased beyond 8. 

Across all step sizes and object orientations, the switching 
controller was successful 75.5% of the time, while the 
convex-only controller was successful only 49.5%. This 
difference was significant according to a Fisher Exact Test 
(p < 10^^). A three-way ANOVA test on the controller 
running time showed a significant effect on the average 
running time of step size (F = 15.3; p < 10"'') and of 
choice of controller {F = 538; p < 10^^). Mean running 
time for the convex-only controller was 258.7sec, while the 
switching controller required a mean of 201.6sec. 

In the concave cube case, across all step sizes and object 
orientations, the switching controller was successful 56.6% 
of the time, while the convex -only controller was successful 
only 8.67% of the time. This difference was significant 
according to a Fisher Exact Test {p < 10^^). The step size 
also had a significant influence on the success rate, according 
to a Chi-squared test (for the switching controUer, x^=342.4, 
p < 10^"*; for the convex-only controller, x^=29.1,p<.016). 



C Varying Object Size 

One critical question to ask is the degree to which object 
size influences the outcome of the search process. To address 
this issue, we presented concave cubes that were 50% larger 
and 50% smaller than the original size reported above. For 
the large cube, across aU step sizes and object orientations, 
the switching controUer was successful 53.8% of the time, 
while the convex-only controller was successful only 6.41% 
of the time. This difference was significant according to 
a Fisher Exact Test (p < 10^^). For the smaUer cube, 
the switching controUer was successful 52.7% of the time, 
while the convex-only controUer was successful 31.6% (Uie 
difference also significant according to a Fisher Exact Test; 
p < 10^^). The latter controller demonstrated a marked 
improvement in performance over the other object sizes. This 
improvement was due in large part to a fortuitous alignment 
between the edge of the cube and the initial positions of 
the fingers. The aUgnment enabled the convex-only search 
process to discover a grasp solution in which the flat regions 
around the corners of the cube were used in the final solution, 
rather than the concavity. 



D. Randomized Object Poses 

In the above experiments, each object was positioned in a 
fixed location and a fixed set of rotations were used (about 
a single axis). In order to examine the robustness of our 
controller over a wider range of initial conditions, we also 
performed a set of experiments in which the pose of the 
object was chosen randomly. We chose the capped cylinder 
and the concave cube with a range of controller step sizes 
(6,7,8, and 9). The object position is selected from a normal 
distribution (standard deviation: 15% of finger time radius) 
and the orientation is selected uniformly from the possible 
3D orientations. 

For each of the two objects, both the convex -only and the 
switching controllers were employed. A total of 100 samples 
were taken for each of the four conditions. For the cylinder, 
the switching controller achieved 86% success, where the 
convex-only controUer achieved 75%. For the concave cube, 
the switching controller achieved a success rate of 49.3%, 
while the convex-only controller achieved a rate of 9.5%. 
For this latter object, of the failures to grasp, more than 
70% were cases in which an equilibrium was successfully 
reached, but either the net force or moment (or both) were 
above the acceptable threshold (e.g., grasping a cube using 
two fingers on the same corner). 

IV. Conclusions AND Future Work 
When grasping and manipulating objects in open environ- 
ments where uncertainties exist in their geometry and pose, 
robust behavior requires the on-line integration of sensory 
data. In particular, one can make use of haptic information 
to guide the search for contact locations that will comprise a 
stable grasp. This search process can be formulated as one of 
gradient descent of a wrench-based error function. However, 
estimation of the gradient of this error function with respect 
to contact movement relies on knowledge of the curvature 
of the surface around the contact point. 

In this paper, we have proposed an approach in which the 
local curvature is estimated to be either convex or concave 
using a pair of spatially distributed probes over the object. 
Given this estimate, the gradient following algorithm can 
appropriately assess the direction in which to displace the 
contact in order to further reduce error. For objects that 
include concave surfaces, our experimental results show a 
substantial performance increase by the proposed algorithm 
over an algorithm in which the surface is always assumed 
to be convex. For convex objects, although the proposed 
algorithm pays a small performance cost, it is not substan- 
tial. These results hold for a range of parameter settings, 
including the magnitude of the search step size. 

Because the search for an appropriate grasp configuration 
follows the gradient of the cost function, the approach suffers 
from the existence of local minima. This was particularly 
clear in case of the concave cube, in which the proposed 
controller failed on 43.4% of the trials, 70% of which were 
cases in which the two fingers were left attempting to grasp 
a single corner of the cube. In general, the existence of 
these local minima is an interaction of the geometry of the 
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object, and the number of contacts involved in the grasping 
process. One can address this issue simply by detecting that a 
local minima has been found, and then restarting the search 
process from a new configuration. In this restart step, one 
could also remove or add fingers, and hence contacts. 



More generally, it is possible to introduce vision-based 
techniques to bias the selection of the initial configuration of 
the fingers. Specifically, we would want such a configuration 
to drop the contacts into a well of the error function whose 
minimum corresponds to an acceptable grasp. Along these 
lines, Coelho, Plater, and Grupen [5] have developed an ap- 
proach in which visual representations are explicitly acquired 
that robustly predict the outcome of a grasp search process. 
Once the models have been learned, it becomes possible to 
use the visual representations in a novel situation to place 
the hand and fingers in a configuration that is often near to 
an acceptable grasp. We are currently working to extend this 
approach to allow grasping of three dimensional objects. 
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